Handling conversation history requires the client to manage state by storing all prior user and assistant messages and sending the full array with every subsequent API request because LLMs are stateless.

One of the most critical concepts in building conversational AI is that LLMs are inherently stateless . They have no memory of previous interactions. When you call the API, it processes only the messages you provide in that single request and then "forgets" everything once the response is generated. This means that to maintain a coherent, multi-turn conversation, the responsibility of managing history falls entirely on the developer's application .

The standard practice is to store the conversation history on the client or server (e.g., in a database or session store). Each time you need the model to generate a new response, you must retrieve the entire relevant conversation history, append the latest user message, and send this full, augmented array of messages in the API request. This includes system instructions, the complete back-and-forth of the conversation, and any previous assistant responses . For long conversations, this can become inefficient, leading to the use of techniques like summarization (compaction) or context injection to manage token usage without losing important context .

Managing Conversation History

Difficulty: 6/10

Topics: conversation state management, token limits, API cost optimization

Scenario Questions

0-2 years experience

1
Suppose you're building a simple chatbot using the OpenAI API. How would you include the previous user messages so the model can keep the context?
2
If you forget to send the full message array on a request, what would the model's response look like and why?
3
Given the model's token limit, how would you decide which parts of the conversation to keep when the history grows?

2-5 years experience

1
You added a feature that trims older messages to stay under token limits, but users report the bot losing context after a few turns. Walk me through how you'd debug this.
2
Explain the trade‑offs between sending the entire conversation each time versus storing a summary on your server and sending that summary with new messages.
3
During a load test, you notice latency spikes when the message array gets large. What changes could you make to mitigate this while preserving context?

5-8 years experience

1
Design a conversation‑history management component for a multi‑tenant SaaS product that uses both OpenAI and Anthropic APIs. How would you handle token limits, cost, and per‑tenant isolation?
2
How would you ensure consistency of conversation state when requests are retried or when you have multiple parallel calls (e.g., streaming responses) to the LLM?
3
Discuss how you would implement a fallback strategy if the API rejects a request because the payload exceeds the model's context window.

8+ years experience

1
At scale, how would you architect a system that supports long‑running dialogues across many users while keeping API costs predictable and allowing future migration to a different LLM provider?
2
What governance and observability mechanisms would you put in place to monitor conversation‑history handling errors and token‑usage anomalies across teams?
3
If you needed to migrate existing conversation logs stored as raw message arrays to a new summarization‑based storage model, how would you plan and execute that migration without breaking active sessions?

Follow-up Questions

❓What would you do if the token count exceeds the model's limit mid‑conversation?
❓How do you balance preserving context with minimizing API costs?
❓Can you describe how you'd test that your history handling works under load?